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USER INTERFACE FOR AUTOMATIC 
RED-EYE REMOVAL IN A DIGITAL IMAGE 

FIELD OF THE INVENTION 
The present invention relates generally to digital photography and more 
specifically to user interfaces used in conjunction with techniques for removing the 
red-eye effect in digital images. 

BACKGROUND OF THE INVENTION 
A pervasive problem in flash photography is the red-eye effect, in which an 
on-camera flash reflects off the back of the eyes of a subject, causing the eyes to 
appear red. The problem is so common that many digital photo-editing applications 
include an automatic or manual red-eye removal feature. Automatic red-eye removal 
is not foolproof, however, and manual red-eye removal can become tedious for the 
user. 

It is thus apparent that there is a need in the art for an improved user interface 
for automatic red-eye removal in a digital image. 

SUMMARY OF THE INVENTION 

A method for removing the red-eye effect in a digital image is provided. An 
apparatus for carrying out the method is also provided. 

Other aspects and advantages of the present invention will become apparent 
from the following detailed description, taken in conjunction with the accompanying 
drawings, illustrating by way of example the principles of the invention. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1A is a high-level block diagram of a digital camera in accordance with 
an illustrative embodiment of the invention. 

Fig. IB is an illustration of the display and some of the input controls of the 
5 digital camera shown in Fig. 1A in accordance with an illustrative embodiment of the 
invention. 

Fig. 1C is a high-level diagram of the memory of the digital camera shown in 
Fig. 1A in accordance with an illustrative embodiment of the invention. 

Fig. 2 is a flowchart of the operation of the digital camera shown in Figs. 1 A- 
10 1C in accordance with an illustrative embodiment of the invention. 

Fig. 3 is an illustration of a simplified digital image in which candidate red- 
eye regions are presented to a user in accordance with an illustrative embodiment of 
the invention. 

Fig. 4A is an illustration of a simplified digital image in which a particular 
15 candidate red-eye region has been selected by a user in accordance with an illustrative 
embodiment of the invention. 

Fig. 4B is an illustration of a simplified digital image in which a particular 
candidate red-eye region has been selected and rejected by a user in accordance with 
an illustrative embodiment of the invention. 
20 Fig. 4C is an illustration of a simplified digital image in which a particular 

candidate red-eye region has been selected and rejected by a user in accordance with 
another illustrative embodiment of the invention. 

Fig. 5 is an illustration of a simplified digital image on which a menu has been 
superimposed in accordance with an illustrative embodiment of the invention. 
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Fig. 6 is an illustration of a magnified and centered view of a selected 
candidate red-eye region within a simplified digital image in accordance with an 
illustrative embodiment of the invention. 

5 DETAILED DESCRIPTION OF THE INVENTION 

Red-eye removal can be made more accurate and effective by automatically 
detecting one or more candidate red-eye regions in a digital image, presenting the 
candidate red-eye regions to a user, and allowing the user to accept or reject 
interactively red-eye removal in individual candidate red-eye regions. A modified 

10 digital image may then be produced in which red-eye removal is applied only to the 
candidate red-eye regions the user has accepted, and the resulting modified digital 
image may be saved. 

Although this detailed description presents the principles of the invention in 
the context of a digital camera, the principles of the invention may be applied to a 

15 variety of other settings, including, but not limited to, digital camcorders, desktop 
personal computers (PCs), workstations, notebook computers, laptop computers, and 
personal digital assistants (PDAs). That is, the invention is applicable to any 
apparatus capable of capturing and/or storing digital images and manipulating them. 
Fig. 1 A is a high-level block diagram of a digital camera 100 in accordance 

20 with an illustrative embodiment of the invention. In Fig. 1 A, controller 105 

communicates over data bus 1 10 with imaging module 115, communication interface 
120, display 125, input controls 130, and memory 135. Optical system 140 produces 
optical images that are converted to digital images by imaging module 115. 
Controller 105 may comprise a microprocessor or microcontroller. Imaging module 

25 115 may comprise an array of photosensors based on charge-coupled-device (CCD), 
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CMOS, or other image sensing technology; an analog-to-digital converter (A/D); a 
gain control; and a digital signal processor (DSP) (not shown in Fig. 1 A). 
Communication interface 120 may be of the hard-wired variety, such as Universal 
Serial Bus (USB) or Firewire (IEEE 1394), or it may be wireless, such as Bluetooth or 
5 IEEE 802.1 1. Communication interface 120 may be used to transfer digital image data 
from digital camera 100 to an external device such as a PC. Display 125 may 
comprise a liquid crystal display (LCD). Input controls 130 may include navigational 
controls (e.g., directional-arrow controls), a menu/"ok" button, a shutter release 
button, or other controls, physical or virtual, for controlling the operation of digital 

10 camera device 100. 

Fig. IB is an illustration of display 125 and some of the input controls 130 of 
digital camera 100 in accordance with an illustrative embodiment of the invention. In 
Fig. IB, digital camera 100 may include a set of navigational controls 145 comprising 
two pairs of opposing directional controls, horizontal directional controls 150 and 

15 vertical directional controls 155, and menu/"ok" button 160. These controls may be 
physical buttons, or they may be virtual buttons on, e.g., a touch-sensitive screen. 
Navigational controls 145 may be used, for example, to navigate among and give 
focus to items on display 125. Menu/"ok" button 160 may be used to call up a menu 
on display 125 and may double as an "ok" button (much like an "enter" key on a 

20 computer keyboard). 

Fig. 1C is a high-level diagram of memory 135 of digital camera 100 in 
accordance with an illustrative embodiment of the invention. In general, memory 135 
may comprise both random access memory (RAM) 165 and non-volatile memory 
170, which may be of the removable variety (e.g., a secure digital or multi-media 

25 memory card). Memory 135 may further comprise red-eye detection logic 175 and 
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red-eye removal logic 180. Red-eye detection logic 175 may detect one or more 
candidate red-eye regions in a digital image and present them on display 125. 
Automatic red-eye detection and removal techniques are well known in the digital 
image processing art. Examples include U.S. Patent 6,278,491 and pending U.S. 
5 Patent Application 10/653,019, both assigned to Hewlett-Packard Company, the 
disclosures of which are incorporated herein by reference. The former reference 
employs face detection; the latter does not. Red-eye removal logic 180 performs red- 
eye removal (e.g., according to the techniques described in the cited references) in the 
candidate red-eye regions of a digital image that a user has accepted. Those candidate 

10 red-eye regions that the user rejects remain unmodified. Essentially, red-eye removal 
involves replacing red pixels with those of a more suitable color where the red-eye 
effect has occurred in a digital image. User interfaces by which the user may accept 
or reject individual candidate red-eye regions will be described in a later portion of 
this detailed description. Red-eye detection logic 175 and red-eye removal logic 180 

15 may be implemented as software, firmware, hardware, or any combination thereof. In 
one embodiment, red-eye detection logic 175 and red-eye removal logic 180 may be 
stored program instructions residing in firmware that are executed by controller 105. 

Fig. 2 is a flowchart of the operation of digital camera 100 in accordance with 
an illustrative embodiment of the invention. At 205, in response to a request from a 

20 user to remove the red-eye effect in a digital image, red-eye detection logic 175 may 
analyze a digital image to detect automatically one or more candidate red-eye regions 
in the digital image. A candidate red-eye region is one that meets the criteria of the 
applicable red-eye detection algorithm. Red-eye detection logic 175 may present the 
candidate red-eye regions to the user on display 125 at 210. At 215, the user may 

25 accept or reject individual candidate red-eye regions. If all candidate red-eye regions 
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are correct, the user may accept red-eye removal in all of the candidate red-eye 
regions by, for example, simply invoking a menu and saving the modified digital 
image (see steps 220 and 225 and Fig. 5). At 220, red-eye removal logic 180 may 
produce a modified digital image by performing red-eye removal in the candidate red- 
5 eye regions that the user has accepted, those the user has rejected remaining 

unmodified. The modified digital image may be saved at 225, after which the process 
may terminate at 230. 

The specifics of how candidate red-eye regions are presented to the user and 
the manner in which the user may accept or reject individual candidate red-eye 

10 regions may vary depending on the application. Figs. 3-6 show some illustrative 
embodiments. However, a wide variety of variations are possible, all of which are 
considered to be within the scope of the invention as claimed. 

Fig. 3 is an illustration of a simplified digital image in which candidate red- 
eye regions are presented to a user in accordance with an illustrative embodiment of 

15 the invention. In Fig. 3, digital image 300 contains a human subject in which two 

candidate red-eye regions 305 have been identified by red-eye detection logic 175. In 
presenting candidate red-eye regions 305 to a user, it is helpful to mark candidate red- 
eye regions 305 in some way. In the example of Fig. 3, each candidate red-eye region 
305 is enclosed within a geometrical figure 310, in this case a rectangle. Other 

20 geometrical figures 310 may be used, or candidate red-eye regions may be marked in 
some other way (e.g., a pointing arrow icon). All candidate red-eye regions 305 may 
be presented to the user simultaneously, as shown in Fig. 3, or the user interface of 
digital camera 100 may be configured to guide the user from one candidate red-eye 
region 305 to the next sequentially, allowing the user to accept or reject each as it is 

25 presented. 
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In embodiments in which all candidate red-eye regions are presented to the 
user simultaneously, it is desirable to provide the user with a way of navigating 
among the candidate red-eye regions 305 and giving focus to (selecting) a particular 
candidate red-eye region 305 to accept or reject. Figs. 4A-4C show two ways in 

5 which this may be accomplished in accordance with illustrative embodiments of the 
invention. In general, horizontal directional controls 150 may be used to navigate 
among the candidate red-eye regions 305 to give focus to (select) a particular 
candidate red-eye region 305. Vertical directional controls 155 may be used to accept 
or reject a particular selected candidate red-eye region 305. 

10 In Fig. 4 A, the user has selected the rightmost of the two candidate red-eye 

regions 305 using horizontal directional controls 150. In some embodiments, the 
selected candidate red-eye region 305 may be in the "accepted" state by default until 
the user indicates otherwise. In the view in which candidate red-eye regions 305 are 
presented to the user on display 125, the candidate red-eye regions 305 may be shown 

15 corrected (with red-eye removal already performed), or the candidate red-eye regions 
may be shown uncorrected until the user has decided which candidate red-eye regions 
305 to accept. To prompt the user that actuating down-arrow control 155 will reject 
the selected candidate red-eye region 305, an icon 405 representing down-arrow 
control 155 may be placed below the geometrical figure 310 enclosing selected 

20 candidate red-eye region 305. Optionally, display 125 may also indicate which 

candidate red-eye region 305 is currently selected (i.e., which has focus). In Fig. 4A, 
an indicator 410 in the form M/N is provided, where the currently selected candidate 
red-eye region 305 is the Mth candidate red-eye region 305 of N total candidate red- 
eye regions 305. In the particular example of Fig. 4A, the selected candidate red-eye 

25 region 305 is the second of two candidate red-eye regions; therefore, indicator 410 is 
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"2/2." In some embodiments, geometrical figure 310 may be drawn more boldly or in 
a different color to indicate which candidate red-eye region 305 is currently selected. 

In Fig. 4B, the user has actuated down-arrow control 155 in the context of Fig. 
4A to reject the selected candidate red-eye region 305 (the rightmost candidate red- 
5 eye region 305 in Fig. 4B). In this example, rejection is indicated by drawing an "x" 
through the geometrical figure 310 enclosing the selected candidate red-eye region 
305. In this case, an icon 405 representing up-arrow control 155 may be placed above 
the geometrical figure 310 enclosing the selected candidate red-eye region 305 to 
prompt the user that pressing up-arrow control 155 will accept the selected candidate 

10 red-eye region 305. The state of acceptance or rejection of a candidate red-eye region 
305 may be indicated in a variety of ways other than a superimposed "x," all of which 
are considered to be within the scope of the invention as claimed. For example, the 
geometrical figure 310 enclosing the candidate red-eye region 305 may be altered in 
some other way, such as a change of shape or color. 

15 In Fig. 4C, a different approach is employed to accept or reject a selected 

candidate red-eye region 305. Two icons 405 representing vertical directional 
controls 155 (up-arrow and down-arrow controls 155) may be placed above and 
below the geometrical figure 310 enclosing selected candidate red-eye region 305, as 
shown in Fig. 4C, to indicate that actuating either vertical directional control 155 (iip- 

20 arrow or down-arrow control 155) will toggle the state (acceptance or rejection) of the 
selected candidate red-eye region 305. 

Optionally, icons representing horizontal directional controls 150 may be 
placed near indicator 410 or near the currently selected candidate red-eye region 305 
to indicate that horizontal directional controls 150 may be used to navigate among the 

25 candidate red-eye regions 305. 



Page 8 of 16 



HP Docket No. 200400322-1 

Fig. 5 is an illustration of simplified digital image 300 on which a menu 505 
has been superimposed in accordance with an illustrative embodiment of the 
invention. Menu 505 may be presented first after the user requests red-eye removal in 
digital image 300, or the user may be brought directly to an "adjust changes" view 
5 such as that illustrated in Figs. 4A-4C. If Menu 505 is presented first, the candidate 
red-eye regions 305 may be presented to the user (e.g., marked as shown in Figs. 3, 
4A-4C, and 5), and the user may accept all of the proposed corrections simultaneously 
by simply executing "Save Changes." Menu 505 also allows the user to save the 
modified digital image after adjusting changes (selectively accepting or rejecting 

10 candidate red-eye regions 305) or to cancel the red-eye removal operation entirely. 
Menu 505 may be invoked and individual commands thereof executed using, for 
example, menu/"ok" button 160. 

In some applications, it may be useful for digital camera 100 (or whatever 
device in which the invention is embodied) to show a magnified view (zoomed in 

15 view) of the selected candidate red-eye region 305, either automatically or in response 
to manual input from the user (e.g., using the zoom lever of digital camera 100). Fig. 
6 shows one example of how this may be done in accordance with an illustrative 
embodiment of the invention. In Fig. 6, the first (leftmost) candidate red-eye region 
305 of simplified digital image 300 (see Figs. 4A-5) has been selected and zoomed to 

20 produce a magnified digital image 600. Optionally, the selected candidate red-eye 
region 305 may be centered automatically on display 125, as shown in Fig. 6. In 
some embodiments, presenting the magnified, centered view of Fig. 6 may be 
performed animatedly. In such an embodiment, as the user navigates to a different 
candidate red-eye region 305, the magnified, centered view of magnified digital 

25 image 600 may also be updated animatedly. 
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The foregoing description of the present invention has been presented for the 
purposes of illustration and description. It is not intended to be exhaustive or to limit 
the invention to the precise form disclosed, and other modifications and variations 
may be possible in light of the above teachings. The embodiments were chosen and 
5 described in order to best explain the principles of the invention and its practical 
application to thereby enable others skilled in the art to best utilize the invention in 
various embodiments and various modifications as are suited to the particular use 
contemplated. It is intended that the appended claims be construed to include other 
alternative embodiments of the invention except insofar as limited by the prior art. 
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